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Abstract—Strategies for the analysis of public policy have been a major focus of interest among social scientists 
during the last decade. This paper introduces a survey research design and perspective which differ from most 
current strategies for public policy analysis. The design, which is basically an ex post facto correlational design, 
employs an unconventional sampling technique to derive policy-impact statements from a small sample. The policy 
analysis perspective of this research is focused on making prescriptive rather than descriptive statements. 

The specific research is the “URBIS” project, a study which is evaluating the impact of automated information 
systems upon the operations of local governments. After briefly discussing the general logic of public policy analysis, 
this paper details the methods and critical issues of this research design. The paper suggests that the URBIS research 
strategy might be viewed as one general model for public policy analysis. 


1, PUBLIC POLICY ANALYSIS 
Generally, public policy analysis attempts to specify the 
relationships between some phenomena which are clas- 
sified as ‘public policy” and some characteristics of the 
environment within which that policy exists. “Policies” 
are the decisions a government unit makes regarding a 
course of action or inaction—to spend X dollars on 
service, not to fluoridate the local water supply, to 
undertake a military action against another country, to 
reorganize a bureaucratic unit. The analysis of such public 
policies can be usefully characterized on two dimensions: 
research perspective and research design. 


Research perspective 

The perspective of empirical public policy research 
normally is descriptive. That is, the committment is to 
describe and explain that which ‘is’, and one of two 
explanatory approaches usually is employed. One ap- 
proach attempts to explain the existence of public policy. 
Some researchers, including Allison[1], Crecine[2], 
Snyder and Paige{3], Moynihan[4], Wildavsky[S] and 
Wanat([6], focus on the processes internal to the 
governmental units which produce policy. Other resear- 
chers who take the same general approach, such as 
Dye[7], Hayes[8] and Long[9], examine the extra- 
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governmental forces which produce policy rather than the 
internal forces. The second approach focuses on the 
impacts of the policy upon affected persons or objects, as 
illustrated by Edelman[10}], Piven and Cloward[12], 
Jencks et al.[{13], Cook[14] and Gibbs |15}. 

At a fundamental level, however, policies are be- 
havioral positions which are consciously taken in order to 
achieve some particular impact. Thus, it is odd that 
empirical public policy analysis is seldom directed 
explicitly to the question: What public policy (P) should 
be undertaken to achieve desired impact (J) given certain 
conditions (Ci, ..., C,)? 

It might be that this question is avoided in social science 
policy research because it is viewed as “normative.” The 
thrust of positivism is to avoid the issue of what ought to 
be. And yet public policy inquiry can, in principle, be both 
empirical and prescriptive. The policy question above can 
be reformulated in a form of the classic scientific mode of 
explanation: If P, the J under conditions Ci, ..., Cy. 


Research design 

Public policy studies employ a variety of research 
methods. The unit of analysis varies in level from the 
individual to the nation, or even to the supra-national 
organization. Data can be collected from direct observa- 
tion, from interviews, from archival or aggregate data, or 
from unobtrusive measures. 

A critical methodological decision in public policy 
research concerns the number of units of analysis. The 
number of units can range from one (the single case study) 
to a handful (comparative case studies) to a very large 
number (the “demographic approach’). Normally, as the 
number of units increases, the ability to generalize and to 
test hypotheses is enhanced. Conversely, small sample 
analyses often are selected where hypothesis generation 
is important, where a richly textured, extensive data base 
is desired, or where limited units of analysis are dictated by 
research feasibility. Trade-offs among these factors are 
characteristic of most decisions regarding the number of 
analytic units. 

Despite these variations in level of analysis, number of 
units of analysis, and sources of data, most examples of 
public policy analysis share a basic research design. They 
are ex post facto analyses.§ The existing relationships 
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among phenomena, conceptualized as variables, are taken 
as given. One measures these variables at one or more 
points in time. Then the analyst assesses linkages between 
the (dependent) variable to be explained and the 
(independent) variable(s) that might explain it. Thus the 
object is to describe/explain those configurations of 
variables which exist in the population (or a representa- 
tive sample) of the units of analysis. 

The experimental research design,t advocated by 
Campbell [22] for evaluating social reforms, is the basic 
alternative to ex post facto designs. Ideally, conditions 
are controlled so that only one independent variable (the 
“treatment’’) is manipulated to examine how it affects the 
dependent variable. This means that the researcher can 
control (usually by random assignment) the exposure of 
units to treatments, limiting the problems of “‘self- 
selection” to treatments and limiting the effect of 
unmeasured variables. More fundamentally, the experi- 
mental design allows the researcher prior control over 
the configuration of variables and the hypotheses being 
examined. 

While experimental designs are desirable, there are few 
examples of their use to analyze public policy as noted in 
Campbell and Stanley [20]. The obvious difficulty of such 
designs is that the natural setting of governments and their 
environments do not allow prior manipulation of system- 
wide variables by the researcher. Even the natural 
occurrence of circumstances facilitating ‘‘quasi- 
experimental” designs is rare and unexpected, as noted 
by Campbell and Stanley [20] and Caporaso and Ross [26]. 
For these reasons most public policy research is ex post 
facto. 


The URBIS research strategy 

This paper presents and analyzes the perspective and 
design of the URBIS [Evaluation of Urban Information 
Systems] project(27] as a strategy for public policy 
analysis. The perspective of the research is to provide 
empirical and prescriptive policy analysis. The research 
attempts to deal imaginatively with the policy-impact 
configurations that exist in current city governments in 
order to specify those policies which could be enacted and 
those impacts which could be expected in a set of “future 
cities.” That is, it aims to identify the mix of policies 
which would facilitate desired impacts, given certain 
characteristics of the task and the environment. 

The design of the research is an amalgamation—of ex 
post facto and quasi-experimental designs; of large 
sample and small sample analysis; of aggregate, survey 
and unobtrusive data. The most unconventional aspect of 
the design involves the sampling strategy, which is 
illustrated fully in the sections below. 

Briefly, the design is as follows. Policies (and policy 
mixes) relating to automated information systems are 
considered to correspond to “treatments” in an experi- 
mental design. Different mixes of these treatment are 
expected to vary in how they affect impacts—that is, the 
local government unit’s decision making, service delivery 
and work environment. In order to better estimate the 
policy-impact relationship within an ex post facto design, 


tExpositions supporting the advantages of experimental re- 
search designs may be found in Fisher[19] and Campbell and 
Stanley [20]. 

¢Some examples of quasi-experiments are Campbell and 
Fiske [21], Suchman{23], Eulau[24] and Campbell and Ross [25]. 


the units of analysis are selected by a factorial sampling 
strategy. This provides a systematic array of treatments 
which, controlling for other conditions, can be analyzed in 
relation to impacts. 


2. THE URBIS RESEARCH PROBLEM 


Conceptual framework for the problem 

The URBIS research seeks to provide practical advice 
to local decision makers regarding choices they can make 
related to automated information systems. The research 
can provide this advice if it discovers the relationships 
between impacts on the one hand, and local government 
policies related to automated information systems on the 
other hand, given certain environmental characteristics. 
Thus, the research involves three major sets of variables: 
controllable policies, policy impacts and policy environ- 
ments. Figure 1 illustrates the pattern of relationships to 
be investigated and some of the major variables in each 
set. 


Controllable policies 

“Policy” means a position that decision makers can 
take with respect to certain behavioral or organizational 
arrangements that the government can control. Thus the 
research seeks to identify the controllable policy variables 
most important to effective automated information 
systems and to specify preferred strategies with respect to 
them. The controllable policies, within which the informa- 
tion system operates, are either organization-based or 
technology-based. 

These policies governing the organization and manage- 
ment of information technology within the municipal 
government are called organization-based policies. Some 
illustrations of these policies include the following: 


Centralization of control over computer hardware 
facilities, system use, and data. 

Management support of the information system. 

User involvement in the information system. 

Technical training of EDP and user personnel. 

Charging for computer usage. 


Technology -based policies are characteristics of the EDP 
technology configuration. They can be viewed as inter- 
vening variables between organization-based policies and 
the policy impacts. They can vary as to pattern and 
amount, but these variables are not subject to such direct 
control and the organization-based policies. For example, 
sharing data or computer equipment ordinarily would not 
occur in the presence of organizational restrictions 
requiring dedication of computer equipment or data. 
Similarly, it would not often occur in the absence of 
centralization of control over facilities and data. These 
technology-based characteristics are considered policies 
because they can be altered over time by the action of 
decision-makers. Some illustrations of technology-based 
policies are: 


Sophistication of computer equipment and software. 

Degree of computer resource sharing. 

Degree of consolidation and integration of data bases 
between functions. 

Extensiveness of use of automated information in a 
government function. 


The local government’s organization-based and 
technology-based policies tend to determine the specific 
policies applied to a given information processing task, 
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Fig. 1. Framework for the research problem: a generalized model for evaluation of information technology in local 
governments. 


such as budget preparation or traffic ticket processing.t 
However, within each city, certain policies may vary 
across tasks, e.g. the degree of user involvement for 
designing automated budget preparation may vary from 
that for designing traffic ticket processing. Thus, although 
task-specific policies are generally identical to the 
organization- and technology-based policies, some 
policies may be found to vary across tasks within the 
same city. 


+See the text discussion below and footnote + on p. 205 for an 
explanation of the information processing task (IPT) as the 
primary unit of analysis in this research. 

$The choice of “impacts” over “outcomes” represents a difficult 
compromise over what and where to measure. One might measure 
effects at the point of interaction between governmental 
service-provider and citizen-client. Or, one might go directly to 
citizens and measure their perceptions of the effectiveness of 
government services. Then, one would attempt to link these 
effects on the variation in these perceptions with variations in 
policies. 

The examination of outcomes in the community has not been 
undertaken, on several grounds: (a) only a few information 
processing tasks, such as utility billing, are directly related to 
citizens and therefore there would not be much to study; (b) in 
most cases, the impacts of an information system would not be 
appropriately evaluated by talking directly to citizens since the 
general citizenry could not accurately comment on the type of 
information received nor the work environment of government 
officials; and (c) a reasonable approximation of the technology’s 
impact on government services (though not on social justice or 
quality of life) can be obtained using information available within 
the government. 


Policy impacts 

“Policy impact” means an effect of information 
technology in the government administration and is 
distinguished from ‘‘policy outcomes,” which are effects 
in the community environment. Ultimately, local govern- 
ments must be evaluated on the basis of their policy 
outcomes, in particular, the social justice and quality of 
their provision of public goods and services to the 
citizenry. However, the URBIS research does not 
directly examine the government-citizen interface. This is 
because information technology currently is used primar- 
ily as a means to improve the internal operations of local 
governments, and therefore, the effects of information 
technology are most directly linked to the government’s 
decision-making and service-delivery activities. 

In particular, information technology might enhance the 
capabilities of the government to make informed deci- 
sions and to manage the provision of public goods and 
services. Improvements in these aspects of government 
activity also might produce improved community out- 
comes. At the level of community outcomes, the role of 
information technology is indirect and inextricably 
combined with the effects of other programmatic factors. 
Thus, the URBIS research framework limits its empirical 
focus to policy impacts, which are more directly linked to 
the use of information technology. This focus on policy 
impacts, rather than ultimate policy outcomes, is modest, 
but seems the most appropriate perspective for examining 
the importance of current information technologies.+ 

Since the government’s policies for information sys- 
tems primarily influence the performance of departments 
and agencies which use information technology, policy 
impacts are determined by observing the relationships 


202 K. L. KRAEMER et al. 


between the policy variables and the behavior and beliefs 
of those using the technology. The impact variables of 
interest include impacts on decision making, service 
delivery and work environment. Impacts on decision 
making are estimated by measuring such factors as the 
range, amount, accuracy, availability, utilization, and 
perceived contribution of information provided to users. 
Impacts on service delivery are examined by estimating 
the costs of performing tasks, the time users take to 
perform tasks, the degree to which automated information 
is perceived to facilitate provision of service, and the 
extent to which users now perform services not per- 
formed prior to automation. Impacts on work environment 
are assessed by measures of user job satisfaction, 
perceived time pressure, task routinization, task supervi- 
sion and promotability requirements. 


Policy environments 

Since the same information system policies are not 
likely to be preferable under all circumstances, the 
research is also concerned with environmental variables 
which measure the non-EDP policy factors that signifi- 
cantly affect information systems. The aim is to discover 
how information system policies are shaped by the social 
and the political environment in which they occur. 

Three major sets of environmental variables seem 
important to the research. The political and administra- 
tive variables include such attributions, and the profes- 
sionalism of elected officials. The community attributes 
include such factors as population size, city growth rate, 
city age, and the socio-economic heterogeneity of the 
population. Extra-community-based policy instruments 
refer to policies outside the city, such as the level, types 
and restrictions attached to federal funding, which might 
shape local policies. 


Prescriptive policy 

The perspective of prescriptive policy analysis is to 
answer the question: What public policy (P) should be 
undertaken to achieve desired impact (I) given conditions 
(C,,..., C,)? The preceding sections have identified the 
variables used to characterize the concepts of policy, 
impact and condition and have explicated the basic nature 
of the linkages between these variables. Even if research 
can accurately and comprehensively characterize the 
linkages, policy prescription is contingent upon a determi- 
nation of the desired impact. For example, expenditure 
control might be a desired impact for financial manage- 
ment in City,. Assume the research reveals that an 
on-line, real-time computer application in the finance unit 
increases the capabilities for budget and expenditure 
monitoring, independent of other measured conditions 
(e.g. centralization, user involvement in design). In this 
case, City, could be advised to implement an on-line 
system in the finance unit. 

It is often the case, however, that policy prescription 
will be more complicated. In some cases, the desired 
impact might be linked to different mixes of policies and 
might obtain only under certain conditions. In such 
situations, prescription will be conditional and probabilis- 
tic, rather than simple and straightforward. It is also likely 
that some desired outcomes will conflict with others. The 
typical objective of political decision makers is to 
balance competing goals rather than to maximize a single 
desired impact. The use of computers by a supervisor to 


monitor the work of subordinates might enhance the 
supervisor’s decision making capabilities; but the subordi- 
nates might perceive that the quality of their work 
environment has deteriorated. When desired impacts 
conflict, this research can specify the trade-offs, given 
certain policy mixes and environmental conditions. But 
the research will not specify either the particular impacts, 
or the balance of impacts which ought to be preferred. 
Thus, in a strict sense, we shall provide information about 
the impacts of policy alternatives rather than prescribe 
choices where impacts are in conflict. 


Problems in common with other policy studies 

As the conceptual framework suggests, this research 
problem has much in common with most policy analyses. 
First, there is a large number of policies to be evaluated. 
To give useful advice to decision makers, one must 
address a significant number of the choices they face. 
Thus, a design for policy analysis should allow for the 
assessment of a number of independent, dependent and 
control variables. Second, most research questions which 
involve multiple independent variables also face problems 
because these variables are highly interrelated. This 
makes an assessment of their independent impact difficult. 
Policy designs should anticipate highly related policies 
(dependent variables) and should attempt to facilitate an 
assessment of their independent effects. Third, this 
research involves multiple levels of analysis—from the 
government system down to the individual user. As with 
many policy analysis problems, a design must be capable 
of systematically relating multiple levels of analysis. 

A fourth common problem is that some of the policies 
to be investigated are rare. The most interesting policy 
innovations sometimes are adopted in a small number of 
places, if they are adopted at all. For example, charging 
for computer usage is an interesting policy. If few 
municipalities utilize direct charges for computer usage, 
then this policy becomes more difficult to evaluate. An 
adequate study design attempts to identify and include 
those sites in which such rare policies have been 
instituted. 

Finally, a design must be feasible in terms of both cost 
and the conditions to which subjects will agree. If we 
wanted to do a policy study of six particular policy 
variables and were unconstrained, we might draw a large 
number of cities at random and then assign a policy 
combination to each city at random. Moreover, we might 
supply each city the resources necessary to pursue the 
assigned policy combinations for a few years, regardless 
of the consequences. Since such an experiment is 
impossible, we must settle for whatever approximates it in 
the natural world. 


3. THE RESEARCH DESIGN 


The URBIS research design addresses these foregoing 
problems through a fusion of research methods that 
constitute a modest design innovation: an ex post facto 
design aimed at policy prescription; a small sample in 
combination with a highly-stratified sampling strategy 
focused on multiple policies; the selection of particular 
cases which have extreme values on the key policy 
variables; the choice of a unit of analysis that facilitates 
multiple levels of analysis; and a mixture of data 
collection and data analysis strategies. The relation among 
these methods can be seen in an overview of the research 
plan. 
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Research plan 

The general plan is a two-phase research program. 
Phase I is a census survey, by self-administered mail 
questionnaires, of chief executives and data processing 
installation managers in U.S. cities with a population of 
50,000 and above.t The Phase I survey instruments 
provide measures that characterize each city’s policies 
towards information technology and its 
political/administrative environment. 

Phase II is a cross sectional study with a sample of 40 
cities and utilizes personal interviews, self-administered 
questionnaires, census and archival data. Cities then can 
be compared “at one point in time” on each variable of 
interest. The cities are selected through a two-stage 
sampling strategy which stratifies the population of cities 
on the major policy relevant variables. Within each city, 
multiple levels of analysis are distinguished including: 
individual users, six information processing tasks com- 
mon to the 40 cities, the operating departments which 
perform the tasks, the data processing installations, and 
the municipality. Data are collected from 50 respondents 
at varying levels in the administrative hierarchy of each 
city government, and the questionnaires provide meas- 
ures of policy impacts on each government’s decision 
making, service delivery and work environment. 


Policy prescription for future cities 

The research design is aimed primarily at permitting the 
researchers to prescribe policies for “future cities” rather 
than to describe the policies followed by the current 
population of cities. Specifically, the research seeks to 
discover the effects of various configurations of policies. 
Therefore the design departs from conventional 
surveys in two ways: (1) it samples at the extremes of 
policies rather than randomly, and (2) it samples primarily 
on policy variables rather than on environmental vari- 
ables. 

A conventional survey, using a random sample of cities, 
would permit accurate description of the population of 
cities in terms of current policies, associated impacts, and 
environmental conditions. Such a design probably would 
be adequate for policy prescription if the sample were of 
sufficient size, if the policies of interest were randomly 
distributed in the population, and if the effects of each 
policy were easily distinguishable. However, large sam- 
ples and conventional random samples do not facilitate 
the intensive study which is required in an area where 
previous policy research is lacking. 

While a small random sample would facilitate intensive 
analysis, it has several drawbacks in distinguishing the 
effects of interesting policy mixes. First, since some 
important policies occur only rarely in the population, 
they would probably be excluded from a small random 
sample. Second, other important policies tend to co-exist 
and therefore the independent effects of such policies 
would be difficult to distinguish with a random sample. In 


+Phase I also gathered data for all counties with a population 
greater than 100,000. This Phase I data for cities and counties is a 
rich data base for analyzing the current state of local government 
computing. Representative analyses include: Kraemer et al.[28} 
and Matthews et al. [29}. This paper, however, is concerned with the 
future cities design in Phase IJ and thus treats the uses of Phase I 
data only in that context. 

This population includes all “cities” identified by the 1972 City 
and County Data Book, plus several townships and cities in 
Puerto Rico not included by the U.S. Census listing. 


order to differentiate the impacts of these policies it is 
necessary to examine cases where the policies do not 
co-exist. To the extent these cases of non-coexistent 
policies are rare, they too would tend to be excluded from 
a small random sample. Third, the impacts of policies 
might be confounded by eivironmental factors. To discern 
the impact of particular policies, it is valuable to maximize 
the variation on the policy variables. Then, if the policy 
does have an impact, it will be discernible from the 
“noise” generated by other factors. Since random 
samples are not likely to maximize the variation, they 
reduce the researcher’s ability to distinguish policy effects 
from these confounding effects. The key to achieving 
these requirements of the research design is the sampling 
strategy. 


Stratification to maximize policy variation 

The research employs a severe form of stratified 
sampling designed to fashion what we can find in the 
natural world toward an array of policies whose impacts 
are important to evaluate. As such, the technique of 
stratification departs significantly from its conventional 
use. Stratification, a well-accepted aspect of sampling 
theory, normally is employed to increase the power and 
efficiency of a sample, as noted in Leege and Francis[17] 
and Kish[30]. However, as sample size becomes smaller, 
the feasibility of extensive stratification normally di- 
minishes. Hence, small samples rarely are stratified on 
more than one to three variables. 

This research stratifies on a substantial number of 
variables (6) despite a small sample size (40), because the 
purpose of stratification here is to assure an adequate 
distribution of cases on variables of major theoretical 
interest, rather than to maximize efficiency. And, it is to 
assure some independence among policies that might be 
highly related in the existing population. 

The population of cities is stratified on the six variables 
deemed to have substantial theoretical impact and policy 
relevance. These six are as follows: 


Degree of automation. 

Sophistication of EDP system. 

Degree of integration. 

User involvement in design of EDP applications. 
Decentralization of EDP. 

Charging for EDP services. 


The stratified sampling process involves four stages. 
First, the universe of cities is defined and bounded to yield 
the sampling population. Second, the population is 
dichotomized on the basis of six policies. Third, sites are 
further partitioned to represent the extremes for each 
policy variable. Finally, cities which maximize the 
differences on the policies are, selected. Each stage is now 
elaborated as it was actually implemented, along with 
specific decisions that must be made to apply the process 
for other studies. 


Definition of the population 

The original universe of cities from which the 40 sites 
were chosen is composed of all 403 incorporated cities 
over 50,000 in population in 1970.4 Values for all 
stratification variables were obtained for all but 16 of these 
cities through the following estimation procedure. De- 
mographic and expenditure data obtained from the U.S. 
Census was regressed on the stratification variables for 
the cities responding to the URBIS census survey (about 
80% of all cities). From these regressions, equations were 


204 K. L. KRAEMER et al. 


formulated to estimate the values of non-responding cities 
for each of the stratification variables. This procedure 
yielded a population of 387 cities with either real or 
estimated values. 

This population was limited further by excluding cities 
with fewer than 12 computer applications. We set this 
lower bound on the degree of automation because most 
policies are relevant to our analyses only when the 
government has a minimal degree of automation. This 
limitation yielded a final population of 310 cities. 


Stratification on policies 

The population of 310 cities was partitioned into 2° = 64 
strata by making each policy variable dichotomous. That 
was done by constructing an index for each variable. All 
values of the index smaller than a selected cutting point 
(dependent upon the distribution of values) on the index 
were treated as low values of the variable, while all values 
above the cutting point were defined as high values. Thus, 
on any variable, the population of cities was split into two 
groups; the six splits define the 64 strata. 

The indexes were constructed from appropriate census 
questionnaire items using weights chosen judgmentally by 
the investigators and scaled from 0 to 100. Cutting points 
were selected from inspection of the distribution of cities 
on each stratification variable. If the distributions had 
been normal or bimodal at each extreme, the median 
would have been chosen as the cutting point. However, 
inspection of the distribution of cities on the URBIS 
stratification variables led to the decision to dichotomize 
each variable at the third quartile. Figure 2 illustrates the 
kind of skewed distribution each index tended to 
approximate. Thus, dichotomization tended to group 
cities on each index into those with common policies and 
those with relatively rare extremes. 

After all six indexes had been constructed and all sites 
labeled 0 to 1 on each of the six indexes, then every site 
was characterized by a sequence of six zeros and ones, 
such as “110010,” which indicates that a particular site is 
in the high half of the population of the first, second and 
fifth variables, and in a low half on the third, fourth and 
sixth variables. The 64 strata were labeled by these 
sequences of 0’s and 1’s. 


+For example, if there are some but not many (say less than 
eight) empty strata, one would employ the above strategy. But, if 
the first randomly selected replicate contained an empty stratum, 
one might reject it and keep selecting replicates until one with no 
empty stratum was found. With still more empty strata, say 8 to 
12, one might systematically search for a non-empty replicate. 

If there are 12 to 24 empty strata. one might best select a 1/2 
replicate (32 strata) or 40 balanced strata at random and then fill 
each empty stratum in it by moving a site closest to that stratum 
into the empty stratum. In this case, it is necessary to define 
“close.” Suppose 001011 is the label of the empty stratum. There 
are six adjacent strata whose lables are obtained by changing (in 
turn) each 0 to a 1 and each | to a 0. Among these six strata we 
would choose the one containing the most sites. Suppose its label is 
001111; then, we would choose the site in that stratum which has 
the smallest value on the fourth index and move the site to the 
empty stratum. Of course, if the most populous adjacent stratum 
were obtained by changing a 1 in the empty stratum to an 0, then 
the site moved would be the one having the largest value of that 
index. 

If more than 25% of the strata is empty, we would probably give 
up the idea of stratifying on six variables and stratify on only five, 
thus partitioning the population into 2° = 32 strata which, hopefully, 
would be mostly occupied. 


Fig. 2. Illustration of distribution approximated by site selection 
index. 


The actual selection of 40 cities depends on the number 
of empty strata. The procedure is illustrated by two 
extreme possible outcomes of the stratification, although 
other outcomes are possible.t If there are no empty strata. 
the preferred procedure is first to select 40 sites by 
choosing one element from each of 40 strata approximat- 
ing a randomly selected 1/2 replicate of a 2° factorial 
design. Specifically, a balanced set of 40 cells is chosen 
randomly from all possible balanced sets. This insures 
that there are 20 sites on the high side and 20 on the low 
side of each of the six variables. However, if fewer than 
40 strata are occupied (for example, 31 has occurred with 
our stratification), then empty cells are filled by locating 
the city closest in space to the empty cell.t 


Further stratification on policy extremes 

If we were primarily interested in determining proper- 
ties of the population of sites, we would simply draw one 
site at random from each of the selected strata. Since our 
primary interest is to determine the relationships among a 
special array of policy configurations, we propose the 
non-random selection of elements from the extreme cases 
within each stratum, The purpose is to increase the 
likelihood that any effects of the policies will be 
discernible, especially given the “noise” from other 
variables that inevitably cloud the measurement of the 
policy-impact relationship. It is more important to know 
whether or not a policy has an effect, than to discern the 
full range of effects possible from minor variations in the 
policy. 

In choosing a site from a stratum, therefore, we would 
Select the one site that has its indexes for the policy 
variable nearest to their extreme values. The process is to 
calculate, for each site, the six differences 


d,, do, d., dj, ds, ds 


Obtained by either subtracting the index from 100 if the 
index is above its median, or subtracting zero from the 
index if the index is below its median. The site whose 
largest d had the smallest value among all largest d°s in 
the stratum would be chosen. 

To illustrate the process for only three variables, 
suppose there are eight sites in the 001 stratum with 
indexes as shown in the left three columns of the 
following Table 1. The three right-hand columns are the 
corresponding d’s. Examining the eight triples on the 
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Table 1. Sample data for site selection “ 


index Values of Sites 99 991 Stratum 


Policy 


Policy 2 


Policy ¢ 


"Assume the medvan inde aluy fer wart poli.y ante equal °O 


right, we find that the fourth one has the smallest 
maximum (19). Hence the fourth site would be chosen 
from that stratum. 

While we suggest the above strategy for selecting sites 
from cells, the URBIS study randomly selected sites from 
cells for two reasons. First, most cells were sparsely 
populated, usually having two to four cities. Second, 
index values within cells often so closely approximated 
one another that extreme value selection was not 
considered useful. 


tThe “object unit of analysis” is the unit whose behavior is to 
be explained. It is distinguished from the “subject unit of 
analysis,” which is the unit whose characteristics/behavior are 
observed or measured. See Eulau [32]. 

{There are about 300 such IPTs covering the full range of 
services provided by most municipal and county governments. 
However, this study does not attempt to deal with all IPTs 
provided by each government. Rather, it is confined to a smaller 
set of six IPTs, or “IPT types’ common to all 40 cities. As 
elaborated in Danziger [33], most information processing tasks fall 
into six general “types” based on their function in relation to 
information. The types are distinguished by the primary modality 
of their use: 


(a) Recordkeeping: Activities which primarily involve the 
entry, updating and storage of data, with a secondary need 
for access; the computer facilitates manageable storage and 
easy up-dating for nearly unlimited amounts of information. 

(b) Calculating /Printing: Activities which primarily involve 
sorting, calculating and printing of stored data to produce 
specific operational outputs; these utilize the computer’s 
capabilities as a high speed data processor. 

(c) Record-Searching : Activities where access to and search of 
data files is of primary importance; by defining parameters, 
relevant cases can be retrieved from a file with speed and 
comprehensiveness; the on-line capability of the computer 
is particularly useful. 

(d) Record-Restructuring. Activities which involve reorgan- 
ization, reaggregation and/or analysis of data; the compu- 
ter is used to link data from diverse sources or to summarize 
large volumes of data as management and planning 
information. 

(e) Sophisticated Analytics: Activities which utilize sophisti- 
cated visual, mathematical, simulation or other analytical 
methods to examine data; the special capabilities of 
computers make possible the manipulation of data about 
complex, interdependent phenomena. 

(f) Process ControL: Activities which approximate a cyberne- 
tic system; data about the state of a system is continually 
monitored and fed back to a human or automatic controller 
who steers the system towards a performance standard; the 
computer’s capabilities for real-time monitoring and direc- 
tion of activities are utilized. 


“Task” as the unit of analysis 

To maximize variation on the policies of interest within 
the cities selected, it is necessary to identify theoretically- 
meaningful units of analysis. A focal point for our 
substantive interests is the city’s automated information 
system. Information systems in cities are inherently 
complex because they are essentially a collectivity—of 
people as users and providers; of information processing 
activities (i.e. data collection, storage, retrieval, manipula- 
tion, summarization and analysis); of equipment and 
facilities comprising a data processing installation; and of 
organization and technology-based policies that control 
the other elements. Because of their complexity, informa- 
tion systems cannot be analyzed as a single entity. Rather, 
they must be analyzed in terms of some elemental units. 

A further complication is that the policies which shape 
these automated information systems can affect various 
levels: (1) individual users or providers, (2) the data pro- 
cessing installation, (3) the user department, (4) the city 
government and (5) intergovernmental units. Therefore, 
the design needs to accommodate these multiple levels of 
analysis. The problem, as conceptualized by Eulau and 
Prewitt(31], is to find a suitable unit of analysis which 
enables us to measure variables at other levels of analysis 
and then reconstruct these variables at the level of the 
analytic unit. While such a procedure requires careful 
specification, it is the most appropriate technique for 
examining phenomena that must be measured at different 
levels of analysis. It is also possible, of course, to collect 
and manipulate data so that more than one primary object 
unit of analysist can be examined. Thus in the URBIS 
design, cities are the primary unit for purposes of 
sampling, individual role-takers are the units providing 
most of the data, and both the “Information Processing 
Task” (IPT) and the types of role-takers are the primary 
object units of analysis. 


The Information Processing Task 

An “information processing task” is a term used to 
signify a particular type of activity undertaken by the 
municipality. The activity has an identifiable function— 
that is, it fulfills a specifiable objective. It explicitly 
involves information processing and might be automated. 
For example, most cities regularly issue a payroll which 
involves translating records of hours worked, pay rates 
and payroll deductions into a payroll check. Thus “payroll 
processing” can be designated an IPT. Similarly, the 
searching of wanted-warrant records on a criminal 
suspect being questioned by a patrol officer in the field is 
the IPT, “patrol officer support.”’+ 

These information processing tasks represent the 
primary unit and level of analysis in the research design. 
They allow meaningful characterization of variations in 
outcomes relative to various mixes of information policies 
with regard to information systems. For example, “patrol 
officer support” may or may not be automated and a 
variety of computer applications could support this task. 
It makes sense, therefore, to evaluate the speed, cost, 
effectiveness, work environment and information availa- 
bility for individuals performing this task as a function of 
the technology-based policies. 

For some theoretical interests, the role-taker (or an 
aggregate type of role-taker) rather than the IPT will be 
the object unit of analysis. Hence, we might compare the 
relationship between measures of perceived quality of the 
work environment and sophistication of EDP for different 
types of role-takers (e.g. clerks vs managers) or for 
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role-takers with different professional orientations and 
different length of job experience. 


Multiple levels of analysis 

The IPT also allows comparability of a task across 
cities. It is evident that city-wide data processing policies 
are identical for all IPTs performed in the municipality. 
Installation-specific policies and technologies generally 
are similar for those IPTs and users serviced by the same 
data processing installation. Operating department 
policies are similar for all IPTs and their users within that 
department. Thus, each IPT has certain IPT-specific 
attributes (e.g. the types of computer applications 
assisting this task), as well as a number of contextual 
variables at the department, installation and government 
levels. The fact that these attributes are common to any 
single IPT allows comparison of the users of a particular 
IPT across municipalities.{ Figure 3 illustrates five levels 
of analysis along with the types of relevant variables at 
each level. 


Types ot variables 


Political, adminiztrative and community attributes 
of the munivipatety 


Municipality 


Information policies af the munic pality (eg, 
centrulization ef control, level ot expenditure on 
£0P, vxtent o* data sharing tetwexn departments, 
etc) 


2 Data Processing 
Instaltations(s) 


Attributes of the jata vroce-3img installation: 
which service the runtcrpality (eq, the kinds 
and aumber of Installations, qualifications of EOP 
personnel, coptisticetion uf hardware 


3 Operating 
Department 


Attributes ct tre user Fr operatina department 
which perform: the infermatian proceseiny tas 
(ec, professionabism “f manogerial gersonnel, 
level of automstinn in "he department, ete ) 


4 Informat von Task-.pecific data processing ool«cies and techii 
Processing ‘ony (eg , the extent to whicn the task i= sute- 
Task mated, the extent of user iivelvemen* i design 

of the autumaled aspecis, the use vf on-fine 
operaticns) 

Task impacts 'e 4 , speed per transaction, ac-u- 
racy of intoemation, perception of quality of 
work environment, agaregited by task? 

5 Individual attributes ct the individual icer ahe rertorms or 
Role-takers, provides the irfermatien Froce:siny Task (Fg, 
or (Isers aye, experience, attitudes tunard computers) 

Task impacte cn user (2g , work ensircnment job 
satisfaction, freedom trom super icer service 
delivery sseed per tranzactien, deci¢iunmaking 
perceived accuracy af intermation use vf auto= 
mated versus manual ti1€=) 

i 


Fig. 3. Levels of analysis for URBIS. 


Data collection 

Each method of data collection has certain 
weaknesses[33]. Therefore, a strategy of the URBIS 
project is to integrate a variety of data collection 
techniques. 

Data will be collected from about 50 respondents in 
each of the 40 cities. URBIS investigators will gather 
documents, conduct semi-structured personal interviews, 
gather lists of users for systematic sampling and deliver 
self-administered questionnaires to each respondent. 


tit is unlikely that IPT findings can be used to generalize about 
all IPTs in a department or city. The behavior and experiences of 
the users of six IPTs are an inadequate basis for making general 
inferences to department or municipal impacts. This is because it 
is unlikely that these users will comprise a representative sample 
of all users of information processing within a department or 
municipality. 

tSee Kish[30, pp. 113-123], for a discussion of systematic 
selection procedures. 

§Techniques similar to the multitrait-multimethod matrix will be 
used for evaluating the scales and indexes. See Campbell and 
Fiske[21]. 


Managers and technical staff of the data processing 
installations will be the sources of most measures of the 
policy variables. Chief executives, departmental mana- 
gers, supervisory staff and line personnel doing the 
relevant IPTs will be the major sources of most measures 
of the dependent policy impact variables—user behavior, 
beliefs and attitudes relating to decision making, service 
delivery and work environment. 

The particular IPT largely will determine the primary 
role-takers who should be sampled. For example, a 
“criminal incident report’? task is utilized by the 
patrolman, detective, precinct captain, police chief and 
others within the police department. Given the particular 
IPT, all those occupying the formal roles involved with 
the task will define the interview population. Each role 
will be weighted by the number of individuals occupying 
the role and by a rough index of the importance of that role 
to the IPT. Systematic sampling procedures then will be 
used to select a sample of specific users within each role.+ 

There are other data sources beyond the self- 
administered questionnaires. First, Phase I questionnaires 
provide a varied data base reflecting the general 
characteristics of the city’s EDP operation, its administra- 
tive and political style, and the chief executive’s 
perceptions of data processing. Second, U.S. census and 
other archival data will be utilized to tap attributes of the 
political, administrative and socio-economic environment 
of the municipality. Third, a number of observations are 
planned, such as obtaining sample reports and computer 
printouts. These will enable us to assess the quality of 
information or types of services provided. Fourth, 
unstructured interviews and observations of the inves- 
tigators during field work will add a mixture of what 
Bruyn([34] calls “participant-observer type” data. 


Data analysis 

Data analyses will consist of two major analytical 
endeavors. The first will be the testing of hypotheses 
suggested by the current state of knowledge and 
experience about information systems. Figure 1 illustrates 
the basic pattern of hypothesized relationships among the 
major classes of variables. 

We shall not collect enough data to examine the entire 
Fig. 1 structure as a single model. Generally, our analysis 
will examine each functional relationship independently 
through a variety of analyses, emphasizing regression 
analysis. By working through the structure systematically, 
some of the hypothesized connections will be supported 
and others will not be supported by the data. The overall 
result should be a considerable specification of the 
structure. This will enhance our understanding of how the 
variables relate to each other and will identify variables 
which seem to be particularly important. 

The second major analytical endeavor will consist of 
inductively examining all of the data in an effort to 
express the general pattern of relationships more eleg- 
antly. This will involve a series of iterations in which we 
try to develop more general constructs by combining 
conceptually related variables into more general scales 
and indexes. This will be accomplished by techniques 
such as factor analysis.§ These more general scales and 
indexes will be interrelated to search for significant 
patterns of relationships not already revealed by the tests 
of specific hypotheses. Thus, in addition to hypothesis 
testing, data analysis will employ alternative techniques to 
develop summary scales and indexes which can be 
interrelated. The patterns of relationships discovered 
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through this process should identify the most significant 
features within the mass of data. 


4, ISSUES RAISED BY THE URBIS DESIGN 


In the evolution of the URBIS research, several 
important design issues were examined extensively. A 
discussion of these issues might further illuminate the 
logic of this design for policy research. The issues can be 
specified in a series of questions which fall into three 
groups: 


A. The risk of extreme stratification. 

(1) If one samples key variables on their extreme 
values, is it possible to assess with accuracy the 
general relationship of those policy variables to 
impact variables? 

(2) Will the design allow one to generalize to the 
population of cities from which the sample is 
drawn? 

(3) Why stratify on so many variables? 

B. Use of a relatively small sample of sites. 

(1) Is the level of analysis upon which units are 
sampled appropriate for the multi-level research 
hypotheses? 

(2) Is the sample adequate for the use of “large 
sample” statistical analysis techniques? 

C. Assessing the impact of policies. 

(1) Given the other factors affecting decision making, 
service delivery, and work environment, can the 
consequences of information system policies be 
identified? 

(2) Is it valid to link information system policies and 
impacts measured at the same point in time? 

(3) Can an ex post facto design overcome the problem 
of self-selection of policies (treatments)? 


A. The logic of extreme stratification 

1. Sampling variables on extreme values. The selection 
of the extreme-values case in each stratum is guided by 
the logic of the design and it directly benefits data 
analysis. It is quite likely that a large number of factors 
affect the dependent variables of decision making, service 
delivery, and work environment. Investigation of the 
linkage between particular policies and the dependent 
variables must deal with the confounding effects (the 
“noise”) of these other factors. It seems reasonable to 
assume that if our cities are near the extremes on each 
policy, any relationship between policy and impact is 
likely to be more discernible. If we are near the extremes 
of each policy and are unable to detect a relationship, we 
have reasonable evidence for inferring that no clear policy 
impact linkage exists. 

In assessing the relationships between major variables, 


+There will not be enough degrees of freedom to rely upon 
individually calculated error variances. 

It is possible that, having selected units near the extreme 
values on policy variables, these units also will be near the 
extremes on the outcome variables. This would further enhance 
our ability to estimate the nature of relationships. 

§All such analyses operate under the assumption that the 
relationship is basically linear. However, even if it were 
curvilinear, a random sample would tend to underestimate the 
linear component relative to an extreme case sample. 

‘Not only should we be able to determine the existence of a 
non-linear relationship, but we should be able to estimate the 
approximate shape of the relationship. 


contingency tables (e.g. 22 tables) and most types of 
analyses employing frequency counts are inappropriate. 
The fact that we have balanced on certain policies insures 
that a 2 x 2 table intended to explore dependence between 
two of the stratifying policies would come out: 


Policy 2 
Low High 
High 


| 10 | 10 | 


Thus, we would detect nothing, even though the two 
policies might be highly correlated. It is possible to do 
simple contingency analysis between a stratified variable 
and an impact measure. But, the differences in propor- 
tions between categories will be somewhat inflated given 
that cases toward the median are under-represented. 

Most variables will be quantified by constructing index 
numbers and scales which approach an interval level of 
measurement. Our typical mode of examining the 
relationship between two variables will be to find the best 
straight line describing the relationship and to judge 
whether the slope is statistically or substantively signifi- 
cant. For statistical significance, the primary technique 
will be the use of a typical (median) error variance, found 
when impact variables are regressed against the stratify- 
ing variables.t For substantive significance, we will rely 
on (1) the relative strength of relationships, (2) the 
proportion of explained variance, (3) an evaluation of the 
general pattern of relationships and (4) our judgments 
regarding the meaningfulness of these relationships from 
a theoretical standpoint. 

It might seem that if variables are represented by 
extreme values, it would be problematic to assess 
relationships across the entire range of values. In fact, the 
extreme-values sample will be particularly good for 
estimating the linear component of a relationship between 
two variables—better than a random sample. This virtue 
obtains because in estimating slopes one does well to 
sacrifice observations near the center in favor of 
observations on the extremes. 

For example, assume that the data for an impact 
variable I plotted against a policy variable P had the 
appearance shown in Fig. 4(a).¢ In essence, our analysis 
would estimate a straight line going through the means of 
the two groups and would declare that the line represents 
the relationship between Impact and Policy.§ 

When extreme cases are selected for analysis, it might 
be that something odd occurs in the center. If, for 
example, the population (from which Fig. 4(a) is sampled) 
has the appearance of Fig. 4(b), then our linear estimate 
would be poor. However, such a poor estimate is quite 
unlikely for two reasons: (1) since we have a census of the 
whole population on a large number of variables, it should 
be possible to discover whether the distribution of 
policies in the population has an odd shape and (2) since 
the cities will not be at the extreme on every variable, 
some cities will have values near the median on any 
particular variable. Due to this partial spread of values, 
the sample is likely to resemble Fig. 4(c). 

A random sample on the same scale might look like Fig. 
4(d); and the whole population might look like several 
variations of Fig. 4(d) imposed together on Fig. 4(c). In 
sum, the sample should have the population well- 
bracketed, and non-linear relationships should be detecta- 
ble during Phase II data analysis, as well as by 
comparison with the Phase I census data.§ 


Policy 1 
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(c) (d) 


Fig. 4. Hypothetical distributions of cities. 


2. Generalizing to the population. While the extreme- 
values sample will be excellent for estimating linear 
relationships, it will be less satisfactory than a random 
sample for estimating parameters of the existing popula- 
tion of cities. It should be clear that the ‘‘future cities” 
perspective is not concerned with estimating population 
parameters. However, when necessary, one can estimate 
them reasonably well by regressing an appropriate 
variable against the stratification or other Phase I census 
variables. 

The basic strategy for estimating population parameters 
from the stratified sample of 40 cities can be illustrated by 
a simple example which assumes linearity of relation- 
ships. Suppose we want to estimate the mean of a variable 
V in the population of all cities (that is, all N cities that 
we have included in our census). We observe V in the 40 
cities of the sample and calculate a simple regression of V 
on the six policy variable indexes, say P,, P2,...., Ps. It 
will have the form: 


V =Aot+ BP: + B2P2+... BePs 


where the B’s are the regression coefficients. Using this 
equation and our knowledge from the census of the P 
values for all the (N-40) cities not in the sample, we 
calculate a V value for all the cities not in the sample. 
Finally we estimate the population mean of V_ by 
averaging the N values of V, 40 of them from the sample 
and N-40 from the regression calculation. 

This would, of course, be the estimate of V for the 
population of existing cities. If we wanted to estimate V 
for the population of ‘future cities” which would result 
from adopting certain of the six policies, then we would 
simply calculate V by giving the P’s the appropriate 
values (perhaps 100 for some and zero for others).t 

3. Why stratify on so many variables? The design 
attempts to stratify on as many variables as possible. The 
sample is not constrained to mirror the distribution of key 


tIf a variable V happens to be definitely non-linear with respect 
to one or more P’s, then we would employ a method more 
complex than simple linear regression. We shall have ample 
opportunity to detect nonlinearity because our selection of 40 
cities will pick up a number of instances in which a P value is not 
far from its median. A brief treatment of non-linear correlation 
and regression is provided by Blalock [36]. 

tThis is often referred to as the “ecological fallacy.” 


variables in the existing population. On the contrary, the 
design aims to identify a special and (probably) unrepre- 
sentative array of configurations. The purpose of this 
unrepresentative sample is to better estimate the indepen- 
dent impacts of policies that are often highly related or 
often rare. 

The stratification of policies is structured by means of a 
2‘ factorial design. In the ideal research world, k would 
equal all those policy variables which seem especially 
important and all those exogenous variables which, 
according to the analysis of the population of cities, seem 
to have a strong effect on the impact of the policy 
variables. The decision that k equals six rather than four 
or ten is based on the constraints of the real research 
world. It is our assessment that approximating a 1/2 
replicate of a 2° factorial design is the maximum variable 
stratification feasible, given the size of the population and 
a 40 city sample. And, sample size has been established by 
the tradeoff between the requirements of adequate data 
gathering for each type of information processing task in 
each city and the availability of resources for research. 


B. Use of a relatively small sample 

1. Is the level of analysis upon which units are sampled 
appropriate for the multi-level research hypotheses? 
Figure 3 characterized the complex environment within 
which the research is embedded. The primary object units 
of analysis, selected on the basis of our theoretical 
interests, are the types of IPTs and the types of 
role-takers. Data will be gathered from many levels of 
analysis. It might seem questionable to sample at a 
different level (city-level policies) than the level at which 
the object units exist (IPT-level or role-level). The brief 
answer is that with careful specification, one can 
construct variables at one level of analysis from data 
collected at other levels. 

Given our interest in policies related to information 
systems, the sampling procedure is based on city-wide 
policies. Most of these policies are likely to have 
comparable impact on all units of analysis. For example, 
sophistication of the EDP system is a contextual variable 
for the operation of each IPT and should affect each IPT 
in the same manner. The strategy does make the 
assumption that in cities where a policy is generally high 
or low, it will be rather high or low for the specific IPTs 
examined. Few of the stratifying variables are, by their 
nature, likely to have substantial within-city variation. 
And of these few variables, there is an even lower 
likelihood that they vary across the particular IPTs 
examined. 

It might be, however, that a stratifying variable does not 
have the same value for each object unit of analysis. For 
instance, in a certain city it might be that the user 
involvement in design is high on four IPTs but low on two 
others. The key point is that data-gathering techniques are 
flexible and allow the generation of a rich data base. Thus 
the level of user involvement will be measured specifically 
for each IPT and thus there will be valid data for the 
relevant analyses. In this manner, the analysis will avoid 
the fallacy of assuming that contextual variables apply 
uniformly to individual cases.t 

2. Is a small sample adequate for large N analytic 
methods? An obvious concern for those familiar with 
survey research is the rather small sample size. It is 
appropriate to question whether a 40-case sample is 
adequate for useful analysis. There are several levels at 
which one can respond to this concern. A broad 
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“quasi-response” is to note the trade-off, in most social 
science research, between intensive case study analysis 
and a Jarge N study. The latter normally allows for more 
rigorous analysis and generalizations at the cost of a 
substantial loss in the richness of data and insight. The 
URBIS design, in the attempt to generate a rich data base 
while expanding the N towards the budgetary limit, is a 
compromise likely to make proponents of each approach 
uneasy. But, to the extent the analytic methods are valid. 
it seems defensible as what Verba[36] calls a “‘disciplined 
configurative approach.” 

The design has been discussed continually in terms of 
40 cases. This is, for many analytic purposes, a conserva- 
tive estimate. The primary object units of analysis are 
normally IPTs, of which there are 6 X 40 = 240 cases. If 
there are differential impacts between IPTs, the number 
(240) will drop. For every key variable that is measured at 
the city-level of analysis and does not vary across IPTs, 
the number will drop. But, only under the most stringent 
assumptions will analysis be limited to N =40. These 
same observations are equally true when types of 
role-takers are the object unit of analysis. 

Even if we take the most limited case where N = 40, it 
1s possible to argue that the sample is not as small as it 
seems. In examining the relationships between variables, 
we shal] look at the effect of other variables. One method 
is the use of one or more control variables.t If we use 
physical controls (for example, if we take high/low EDP 
expenditure as a third variable in contingency analysis), 
the number of cases in the cells is likely to diminish 
quickly. 

However, our techniques for analysis normally will 
employ more complex mathematical and_ statistical 
methods of control. If correlation techniques are used, 
multiple and partial correlations provide _ statistical 
methods to control or include third, fourth, and so on, 
variables without such severe sample size problems. In 
most cases, we expect to examine the relationships 
between variables by means of regression analysis. It is 
obvious that careful and insightful interpretation of 
regression coefficients will be required, particularly 
where multicollinearity or degrees of freedom become 
problematic. But, we argue, cautious analysis of such a 
sample is quite valid. Moreover, the benefits of sampling 
extreme-values cases must be reiterated. Since the more 
extreme scores are more useful in linear estimation, it is 
“as if’ we were analyzing a substantially larger sample 
than the actual number of cases implies. In sum, it seems 
reasonable to assert that even where the sample is small, it 
is data rich and it is able to support rigorous analytic 
techniques. 


C. Assessing the impact of policies 

1. Given other factors affecting decision making, service 
delivery, and work environment, can the consequences of 
information policies be identified? This research assumes 
that organization-based and technology-based policy 
instruments related to information systems are only a 
subset of the factors affecting the quality of decision 


+On the techniques of elaboration, see Rosenberg[38). 

+The use of within-government respondents rather than citizens 
means that the data will not explicitly measure the direct benefits 
or costs to those citizens. However, it seems reasonable that if 
IPT users are clearly affected by information policies, it is 
possible to infer how citizen-clients might be affected. 


making, service delivery and work environment in local 
governments. How are we to assess the impact, if any, of 
the information system variables among this broader 
array of factors? Specifically, are the survey data 
adequate to determine whether policy-impact relation- 
ships do exist? Several aspects of the Phase II design 
attempt to be responsive to this problem. 

First, we assume that the “signals” indicating a 
policy-impact relationship will be augmented by examin- 
ing cases near the extremes on each policy. Second, 
impact measures are generated by interviewing those with 
the most direct experience of the information system— 
those governmental actors who are closely related to each 
IPT. Thirdly, by focusing the respondents’ attention to 
particular Information Processing Tasks, more valid 
measures of performance can be obtained. Since the 
respondents will be those most familiar with the 
information system and since the question stimuli will be 
quite focused, their responses should be near the 
policy-impact interface.t 

The fourth characteristic of the design that facilitates 
assessment of the policy-impact relationship also con- 
cerns the restriction to a few particular IPTs. With a 
limited number of well-defined foci, the research can be 
sensitive to other major, non-information-system vari- 
ables. Insight into each particular IPT enables us to 
identify plausible rival hypotheses to the impact of 
information system policies. By measuring these other 
variables, the analysis can control, or at least account for, 
their effect. If, for example, one examines the IPT of 
“patrol officer support” involving search of wanted 
warrant records, one indicator of decision making would 
be the proportion of times an officer, having stopped a 
motorist, checks the warrant file. While various charac- 
teristics of the police information system are likely to 
affect this measure, other variables are also quite 
important. For example, the existence in City, of a 
well-articulated standard operating procedure (e.g. con- 
cerning certain conditions under which the wanted 
warrant file should be searched) might have a substantial 
influence on the behavior of officers. In sum, we suggest 
that it is possible to assess the relationship between 
information policies and impacts, despite a complex, 
“noisy” environment. In particular, the design employs 
careful measurement of responses by those most know- 
ledgeable about well-defined Information Processing 
Tasks. 

2. Is it valid to link information system policies and 
impacts at one point in time? Most cross-sectional 
analyses, including the Phase II design, must assume that 
the variables related on the basis of current values have 
had a rather similar relationship through time. One might 
expect, for example, that a city with a large number of 
on-line terminals decentralized to user departments would 
be characterized by more extensive use of automation 
than a city with batch operations. But it might be that in 
one city many on-line terminals were only recently 
installed and are not yet affecting extensiveness of use. 
One can imagine instances where the current configura- 
tion of policy variables affecting impact variables is 
different from that at earlier time points. In such 
instances, inferences about policy-impact linkages based 
on current patterns might be misleading. This problem is 
especially important to research examining local govern- 
ment EDP, because some EDP environments evolve 
rapidly. Moreover, there normally is a time-lag between 
the implementation of an EDP policy or technology and a 
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measurable impact on decision making or service 
delivery. 

Still, there is good theoretical and empirical evidence 
supporting the assumption that, ceteris paribus, one of the 
best predictors of what a city currently is doing is what it 
did several years ago. First, most policy seems to develop 
incrementally.t Even with the changeability that charac- 
terizes data processing, it seems that within city variation 
over time will be less than between city variation. This is a 
safer assumption if, as Phase II design proposes, we get 
out on the extremes of the various policies to be 
investigated. Cities currently at opposite extremes on one 
of the policies are likely to have differed in the same way 
several years ago. Second, focusing on specific IPTs and 
interviewing EDP and user personnel facilitates sensitiv- 
ity to those situations where there has been a significant 
alteration in the policy-impact linkage during the recent 
past. Third, Phase I census data can be compared with 
Phase II data to locate radical changes which have 
occurred during the past two years.t 

3. Can an “ex post facto” design overcome the problem 
of self-selection of policies (treatments)? One of the 
major characteristics of an experimental design is the 
ability to randomize the introduction of treatments to 
subjects (the units of analysis). With this capability, the 
confounding effect of any other variable is randomly 
distributed. In contrast, most macro-level social science 
research begins with natural settings, where the introduc- 
tion of the treatment to subjects cannot be controlled. The 
problem is that there are likely to be important interaction 
effects between the treatment and other characteristics of 
the unity of analysis. In particular, each unit (in our 
framework, each city) has selected and maintained those 
treatments (those information system policies) which are 
most congenial with other important variables (other 
important characteristics of the local political system and 
its environment). Thus, one expects to find systematic 
effects between treatments, other measured variables, and 
other unmeasured variables. The complexities of these 
interactive effects make analysis quite difficult. 

It is reasonable to assume, for example, that a city is 
more likely to adopt and maintain data sharing arrange- 
ments if other conditions are congenial—if it has a 
centralized computer installation, if operating depart- 
ments are not independent and territorial, if the chief 
executive stresses cooperative arrangements. Where 
these conditions are not present, the survival and/or 
success of data sharing is problematic. 

it must be stressed that this design does not fundamen- 


tally alter the problems related to treatment effects. 


Although the policy configurations are arrayed so that 
they tend to be uncorrelated, they are self-selected in each 
city. Moreover, there are no special controls against the 
confounding effects of variables other than the stratifying 
variables. Imperfect control of these other variables is 
attempted, as in most multi-variate social science 
research, by the use of statistical controls. While the 
design cannot overcome treatment-city interactions, it 


tThis is well documented in Lindblom[39], Cyert and 
March[40], Allison[1], Wanat [6], Crecine[2], and Davis et al. [41]. 

It would be desirable to extend Phase If into a longitudinal 
(panel) study because the design currently does not provide 
evidence for strong statements about change dynamics or lagged 
effects. But, cross-sectional measurement and analysis are 
appropriate for developing valid generalizations about the linkages 
between policy and impact variabies. 
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does seem that such effects are dealt with more 
adequately than in most natural setting analyses because 
of our attempt to select cases which independently 
distribute the major policy variables. 

It should be added that self-selection will have 
predictable and interesting effects. Because cities select- 
ing their own policies are more likely to find them 
congenial, the treatment-city interactions will be signifi- 
cantly reduced and the main effects correspondingly 
exaggerated. In other words, an experiment that randomly 
assigned policies would often assign a policy to a city 
where it is not compatible. This would tend to reduce the 
observed main effects. Thus, the relationships we find 
between policies and outcomes will be stronger than what 
would be found if the policies were randomly assigned. 


5. A GENERAL DESIGN FOR POLICY RESEARCH 


Experimental designs often are advocated as most 
appropriate for evaluation and policy research. While the 
internal validity of the experimental design is attractive to 
policy scientists, there seem to be several drawbacks to 
the approach. Most significantly, it is usually not feasible 
to investigate the more significant policy questions within 
their natural settings as an experiment. Although some 
investigators have been especially perceptive in identify- 
ing situations adaptable to natural experiments, these 
policy studies remain an exception. 

The difficulty of utilizing the experiment to evaluate 
social policies has resulted in the common use of ex post 
facto designs. However, studies which attempt to 
describe the relationships between public policies and 
their antecedents or their hypothesized consequences 
have other difficulties. While normally less obtrusive and 
more widely applicable than experiments, these descrip- 
tive studies often fail to identify policy-impact relation- 
ships. We believe this is partly due to an over-concern 
with describing the present, resulting in infrequent 
examination of cases with rare or extreme policy 
differences. 

In order to incorporate the evaluation of rare policies, 
the “future cities” design examines a sample which looks 
quite unlike the present population. The sample is stratified 
on each major policy of interest in order that its 
independent impacts might be better estimated. The 
extreme cases of different policies are selected to amplify 
the signal of any policy-impact relationship. Also, the 
sample is large enough to allow the utilization of large 
sample data analysis strategies; yet it is small enough to 
facilitate the collection of an intensive and rich primary 
data base. While the future cities survey design is not 
without problems and risks, it is a potentially useful 
approach to other research questions. Finally, it should be 
noted that the logic of the future cities design is not 
restricted to policy analysis. In most research problems 
where the substantive interest is to assess hypothesized 
relationships between certain key variables, the strategy 
of extreme stratification and extreme cases might be 
employed. 
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